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FINAL  REPORT 


ON 

"FREE  BOUNDARY  CONTROL  OF  MARKOV  PROCESSES" 
Research  Project  AFOSR  88-0183 
Principal  Investigator  Michael  Taksar 

April  13,  1992 


This  work  is  a  continuation  of  the  research  conducted  under  the 
auspices  of  AFOSR  grant  87-0278. 

The  objective  of  this  project  was  to  further  develop  the  theory 
of  controlled  diffusion  processes  and  its  applications.  The  main  line 
of  research  deals  with  control  of  dissipative  stochastic  systems  which 
can  be  viewed  as  a  model  of  an  aircraft  under  uncertain  wind 
conditions  or  perturbation  in  the  mechanical  unit. 

The  principal  investigator  visited  the  French  National  Institute 
in  Information  and  Automatic  Control  (INRIA)  doing  joint  research  with 
French  scientists  on  multidimensional  singular  control  theory.  This 
visit  was  supported  by  NATO  international  exchange  grant  awarded  in 
conjunction  with  the  AFOSR  grant.  In  the  previous  years  the  theory  of 
singular  stochastic  control  was  being  developed  mainly  for  the 
stationary  or  infinite  horizon  case.  There,  the  time-parameter  does 
not  enter  explicitly  the  corresponding  Hamilton-Jacobi-Bellman 
equation.  Non-stationary  analysis  for  the  singular  control  was  done 
so  far  only  for  the  one-dimensional  case.  It  appears  that  the  recent 
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work  of  the  principal  investigator  at  INRIA  will  enable  to  make 
progress  in  the  finite-horizon  multidimensional  singular  control 
theory . 

Another  avenue  of  research  deals  with  general  linear  additive 
controls  and  general  convex  control  cost  structure.  In  such  problems 
the  number  of  control  functionals  m  is  not  necessarily  the  same  as 
the  dimension  n  of  the  process,  and  each  coordinate  of  the  process 
is  controlled  not  by  one  functional  rather  by  a  linear  combination  of 
all  m  functionals.  The  cost  of  instantaneous  displacement  in  this 
model  is  proportional  to  the  length  of  displacement  with  coefficient 
of  proportionality  dependent  on  the  direction.  The  proportionality 
coefficients  in  turn  are  determined  by  a  norm  in  the  m-dimensional 
space  (the  so  called  Minkowski  norm).  The  unit  ball  in  this  norm  can 
be  any  convex  set  containing  zero.  Control  problems  with  such  cost 
structure  arise  in  inventory  theory  and  network  of  queues,  when  one 
uses  diffusion  approximations  for  the  underlying  stochastic  processes. 
Menaldi  and  Taksar  are  currently  completing  the  second  phase  of 
research  on  singular  control  with  general  convex  cost  structure.  This 
is  a  large  problem  which  has  been  only  partially  solved. 

There  is  an  important  issue  in  stochastic  control  problem, 
namely:  -  why  one  usually  minimizes  the  expected  cost?  While  there 
are  other  criteria,  this  one  seems  to  be  the  most  useful  and  the 
easiest  to  deal  with.  It  turns  out  that  for  an  ergodic  problem,  the 
one  which  deals  with  expected  average  cost,  this  criterion  is  much 
stronger  than  one  could  anticipate.  Namely,  the  policy  which  is 
optimal  in  the  mean  in  the  ergodic  problem  is  also  optimal  in 
probability  and  almost  surely.  The  latter  means  that  any  other  policy 
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yields  larger  average  cost  almost  surely.  This  phenomenon  is  shown  in 
the  paper  by  Presman,  Rotar,  and  Taksar. 

Significant  progress  was  achieved  in  studying  diffusion 
approximation  for  controlled  stochastic  systems.  In  a  series  of 
papers  by  the  principal  investigator  and  collaborators  different 
controlled  stochastic  systems  (queueing,  inventory,  manufacturing)  in 
heavy  traffic  were  considered  and  their  limiting  behavior  was  studied. 
It  was  shown  that  in  each  case  there  exists  an  appropriate  singular 
control  problem  solvable  in  closed  form  which  provides  an 
asymptotically  optimal  policy  for  the  original  system.  These  results 
bring  both  justification  of  diffusion  approxiamtion  for  controlled 
queueing/inventory  models  as  well  as  a  practical  way  of  dealing  with 
such  models  when  they  cannot  be  solved  explicitly. 

Two  graduate  students  were  supported  by  the  grant.  Paulo  Santana 
was  supported  in  his  graduate  studies  by  this  grant  at  Florida  State 
University.  He  has  successfully  completed  his  research  on  the 
problems  associated  with  singular  control  on  a  finite  time  interval. 

He  has  completed  the  writing  of  his  thesis  "Finite  Horizon  Singular 
Control  and  a  Related  Two-Person  Game".  He  defended  his  thesis  in 
Spring  of  1988.  In  his  work  he  has  rigorously  proved  the  link  between 
the  optimal  singular  control  on  a  finite  time  interval  and  an  optimal 
stopping  game  of  two  persons  with  opposite  interests.  Charu 
Srivastava  was  supported  in  her  graduate  studies  at  the  State 
University  of  New  York  at  Stony  Brook.  She  is  currently  finishing  her 
thesis  "Optimal  Policices  in  Queueing  and  Manufacturing  Models"  which 
she  expects  to  defend  in  Spring  1992. 


The  results  of  the  research  conducted  under  the  auspices  of  AFOSR 


were  presented  at  the  Eighth  and  Ninth  International  Conference  on 
Analysis  and  Optimization  of  Systems,  Antibes,  France,  June,  1988,  and 
June  1990  and  at  the  27th  IEEE  International  Conference  on  Decision 
and  Control,  Austin,  Texas,  1988,  and  at  the  International  Conference 
on  Control  and  application  in  Jerusualem,  Israel,  April,  1989,  and  at 
the  joint  TIMS-SOBRAPO  International  Conference,  Rio  de  Janeiro, 
Brazil,  1991.  Below  is  a  list  of  work  conducted  under  the  auspices  of 
the  AFOSR  grant  88-0183. 


1.  KANTOROVICH'S  FUNCTIONALS  IN  THE  SPACE  OF  MEAURES  by 
S.T.  Rachev  and  M.  Taksar  (submitted  to  Stochastics ) . 

The  paper  studies  the  problem  of  finding  measures  on  Cartezian 
product  with  given  marginals  in  such  a  way  that  a  specified  integral 
functional  atains  the  minimal  value.  It  analyzes  relationships 
between  different  functionals  and  associated  distances  between  the 
marginal  distributions.  The  latter  problem  relates  to  the  classical 
Monge-Kantorovich  problem  of  translocation  of  masses,  to  the  problem 
of  the  best  allocation  policy,  the  problem  of  optimal  training  of 
recruits  for  the  set  of  jobs,  etc. 

2.  OPTIMAL  CORRECTION  PROBLEM  OF  A  MULTIDIMENSIONAL  STOCHASTIC 

SYSTEM  by  J.L.  Menaldi  and  M.  Taksar  Automatica .  25  (1989),  223-232. 

We  consider  a  dissipative  system  under  uncertainty  governed  by  a 
multidimensional  diffusion  process.  We  are  allowed  to  correct  the 
position  of  the  system  by  a  means  of  additive  functional.  The  cost  of 
correction  is  proportional  to  the  length  of  the  displacement  of  the 
position  of  the  system  induced  by  the  functional  with  coefficients 
dependent  on  the  direction  of  displacement.  The  objective  is  to 
minimize  the  expected  cumulative  cost  associated  with  the  position  of 
the  system  and  the  amount  of  control  exerted.  It  is  shown  that 
Hamilton-Jacobi-Bellman' s  equation  of  the  problem  has  a  solution  which 
corresponds  to  the  optimal  cost  of  the  problem.  An  existence  of 
optimal  policy  is  proved. 
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3.  AN  ASYMPTOTIC  ANALYSIS  OF  HIERARCHICAL  CONTROL  OF 
MANUFACTURING  SYSTEMS  BY  H.M.  Soner,  J.  Lehoczky,  S.  Sethi  and  M. 
Taksar,  Mathematics  of  Operations  Research,  16  (1991),  3,  596-608. 

This  paper  presents  an  asymptotic  analysis  of  a  hierarchical 
manufacturing  system  with  machines  subject  to  breakdown  and  repair. 

The  machine  fluctuations  are  much  faster  than  the  accumulation  and 
discounting  of  costs,  and  this  gives  rise  to  a  limiting  problem  in 
which  the  stochastic  machine  availability  is  replaced  by  the 
equilibrium  mean  availability.  The  value  function  for  the  original 
problem  converges  to  the  value  function  of  the  limitiang  problem.  The 
limiting  problem  is  computationally  tractable  and  sometimes  has 
a  closed  form  solution. 

4.  EXACT  BOUNDS  FOR  THE  DIFFERENCE  BETWEEN  A  MEAN  AND  A  MEDIAN 
by  M.  Taksar  (technical  report). 

We  show  that  if  a  distribution  has  the  second  moment  then  the 
maximal  distance  between  its  mean  and  its  median  does  not  exceed  its 
standard  deviation  o  .  If  we  restrict  ourselves  only  to  the 
distributions  with  unimodal  densitites  then  the  same  difference  does 
not  exceed  <t4  . 6  .  In  both  cases  the  bounds  are  sharp.  The  latter 
problem  is  reduced  to  the  problem  of  finding  minimal  variance  in  the 
class  of  al]  nonnegative  random  variables  with  monotone  densities  and 
with  means  equal  to  1. 

5.  SINGULAR  CONTROL  FOR  MULTIDIMENSIONAL  GAUSSIAN  PROCESSES  by 
J.L.  Menaldi,  M.  Robin  and  M.  Taksar,  Mathematics  of  Controls,  Signals 
and  Systems,  5  (1992),  93-116. 

This  work  extends  the  results  of  the  singular  control  theory  in 
two  directions.  Firstly,  it  considers  the  underlying  process  with 
nonconstant  drift  coefficients.  Secondly,  it  considers  ergodic 
optimality  criteria,  rather  than  usual  discounted  criteria. 

6.  EXPLICIT  SOLUTION  OF  A  GENERAL  CONSUMPTION/PORTFOLIO  PROBLEM 
WITH  SUBSISTENCE  CONSUMPTION  AND  BANKRUPTCY,  by  S.  Sethi,  M.  Taksar 
and  E.  Presman,  to  appear  in  Journal  of  Economic  Dynamics  and  Control. 

This  paper  gives  a  closed  form  solution  for  a  general 
continuous-time  single  agent  consumption  and  portfolio  decision 
problem  with  subsistence  consumption.  The  analysis  allows  for  general 
continuously  differentiable  concave  utility  functions.  The  model 
takes  into  consideration  that  consumpti'^n  must  be  no  smaller  than  a 
given  subsistence  rate  and  that  bankruptcy  can  occur. 
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7.  INFINITE  HORIZON  INVESTMENT  CONSUMPTION  MODEL  WITH 
NONTERMINAL  BANKRUPTCY,  by  S.  Sethi  and  M.  Taksar,  Journal  of 
Optimization  Theory  and  Applications,  74  (1992). 

This  paper  solves  a  general  continuous-time  consumption  and 
portfolio  decision  problem  for  a  single  agent  for  whom  there  exists 
upon  bankruptcy,  a  possibility  of  recovery  from  his  bankruptcy.  The 
main  contribution  of  the  paper  is  in  the  modeling  of  the  recovery 
process.  Moreover,  it  is  shown  that  the  model  with  recovery  has  a 
one-to-one  correspondence  with  the  model  with  terminal  bankruptcy 
treated  in  the  literature. 

8.  SKOROHOD  PROBLEMS  WITH  NONSMOOTH  BOUNDARY  CONDITIONS,  by 
M.  Taksar,  to  appear  in  Journal  of  Computational  and  Applied 
Mathematics . 

A  stochastic  differential  equation  in  a  region  with  oblique 
reflection  on  the  boundary  is  considered.  The  existence  of  a  weak 
solution  to  the  Skorohod  problem  in  the  region  is  proved  without 
requirement  on  the  smoothness  of  the  boundary  or  the  reflectdion 
vector-field . 

9.  OPTIMALITY  IN  PROBABILITY  AND  ALMOST  SURELY  FOR  AN  INFINITE 
HORIZON  LINEAR  REGULATOR  PROBLEM  by  E.  Presman,  V.  Rotar  and  M. 

Taksar,  to  appear  in  Stochastics . 

We  consider  a  classical  linear  regulator  stochastic  control  model 
in  which  the  objective  is  to  minimize  the  expected  limiting  average 
(ergodic  in  another  terminology)  cost.  We  show  that  the  optimal 
stationary  Markov  policy  u*  for  this  problem  is  also  optimal  in  a  much 
stronger  sense.  Namely,  it  minimizes  the  limiting  average  cost  almost 
surely,  thus  being  uniformly  better  than  any  other  policy. 

We  also  compare  u*  with  any  family  of  policies  u^ .  It  is  shown 
that  probability  that  the  averge  cost  on  [0,T]  associated  with  u* 
exceeds  the  one  associated  with  u^  by  e,  converges  to  zero  as  T-^®. 

10.  DIFFUSION  APPROXIMATION  FOR  GI/G/1  CONTROLLED  QUEUES  by  E. 
Krichagina,  and  M.  Taksar,  submitted  to  Queueing  Systems. 

A  queueing  model  is  considered  in  which  there  is  a  cost  associated 
with  the  length  of  the  queue.  The  cost  is  represented  by  a  holding 
cost  function  h.  A  controller  can  increase  the  service  rate  incurring 
the  service  cost  proportional  to  the  ip'Meased  rate  with  coefficient 
1.  The  objective  is  to  minimize  the  total  expected  discounted  cost. 

When  h  and  1  are  small  the  control  problem  can  be  approximated  by 
a  singular  stochastic  control  problem  for  Brownian  motion,  namely,  the 
so-called  "reflected  follower"  problem.  The  optimal  policy  in  this 
problem  is  characterized  by  a  single  number  z*  so  that  the  optimal 
process  is  a  reflected  diffusion  in  [0,z*]. 
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11.  A  REGENERATIVE  GENERALIZED  ELASTIC  SCREEN  PROCESS  AS  A  HEAVY 
TRAFFIC  LIMIT  FOR  QUEUES  WITH  OPTIMAL  INTERRUPTIONS  by  0.  Kella  and  M. 
I.  Taksar,  submitted  to  Mathematics  and  Operations  Research. 

We  consider  a  queue  in  which  the  server  is  available  for  primary 
customers  for  a  random  number  of  busy  cycles,  after  which  he  leaves 
for  a  random  amount  of  time  as  soon  as  the  system  becomes  empty. 

Under  appropriate  normalization  we  establish  a  heavy  traffic  limit 
which  turns  out  to  be  a  regenerative  generalized  elastic  screen 
process  (RGESP)  with  random  jumps  or  linear  parts  of  the  trajectories. 
These  jumps  or  linear  parts  occur  when  the  local  time  at  zero 
accumulates  to  a  certain  random  level.  We  compare  the  stationary 
distribution  for  the  RGESP  with  the  corresponding  one  for  the  M/G/1 
queue,  where  for  both  explicit  expressions  are  obtained,  and  show  that 
the  results  are  consistent. 

12.  PRODUCTION  CONTROL  IN  A  FAILURE-PRONE  MANUFACTURING  SYSTEM: 
DIFFUSION  APPROXIMATION  AND  ASYMPTOTIC  OPTIMALITY  by  E.  Krichagina,  S. 
Lou,  S.  Sethi  and  M.  I.  Taksar,  to  appear  in  Annals  of  Applied 
Probability. 


We  consider  the  problem  of  controlling  the  production  rate  of  a 
single  machine,  single  product,  stochastic  manufacturing  system  in 
order  to  minimize  the  total  discounted  inventory/backlog  costs.  The 
demand  has  two  components:  one  deterministic  with  a  constant  rate  d 
and  the  other  stochastic  with  random  demand  batches.  Under  the  heavy 
loading  (or  heavy  traffic)  condition,  i.e.,  when  the  average 
production  capacity  is  close  to  the  average  demand,  control  problem 
can  be  solved  explicitly.  The  solution  is  then  interpreted  in  terms 
of  the  actual  manufacturing  system  and  a  control  policy  for  this 
system  is  derived.  We  prove  that  the  resulting  policy  is  nearly 
optimal  under  the  heavy  traffic  condition. 

13.  DOUBLE  BAND  POLICY  FOR  STOCHASTIC  MANUFACTURING  SYSTEMS  IN 
HEAVY  TRAFFIC  by  S.  Lou,  E.  Krichagina  and  M.  I.  Taksar  (technical 
report ) . 


We  consider  the  control  of  a  manufacturing  system  producing  one 
product.  The  demand  for  this  product  is  random.  The  manager  can 
produce  the  product  at  a  fixed  rate  r,  or  to  stop  the  production. 

Each  time  the  production  is  resumed,  a  setup  cost  must  be  paid.  To 
prevent  the  backlog,  which  is  not  allowed,  the  manager  can  buy  the 
product  from  outside  vendors,  paying  a  fixed  order  cost  and  a  variable 
cost.  There  is  also  a  linear  inventory • holding  cost.  The  objective 
is  to  minimize  the  total  expected  discounted  cost.  Under  heavy 
traffic  condition,  i.e.,  when  the  production 
average  demand,  this  problem  is  approximated 
problem  for  Brownian  motion,  which  is  solved 


capacity  is  close  to  the 
by  an  impulse  control 
explicitly. 
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14.  ASYMPTOTICALLY  OPTIMAL  POLICIES  FOR  CONTROLLED  QUEUES  IN 
HEAVY  TRAFFIC  by  E.  Krichagina  and  M.  I.  Taksar  (technical  report). 

We  consider  GI/G/1  queueing  system  operating  in  heavy  traffic. 
There  is  a  proportional  holding  cost  with  coefficient  h.  A  controller 
can  increase  the  service  rate  (thus  moving  the  system  into  light 
traffic)  incurring  the  cost  proportional  to  the  increased  rate  with 
coefficient  1.  Given  the  discount  rate  p  the  objective  is  to  minimire 
the  total  expected  discounted  cost.  We  show  that  when  the  parameters 
S  associated  with  heavy  traffic  and  &=h/l  are  small  and  p  has  an 
appropriate  order  this  control  problem  can  be  approximated  by  a 
singular  control  problem  for  Brownian  motion,  namely,  reflected 
follower  pro'^lem. 


Research  in  Progress 

1.  MULTIDIMENSIONAL  SINGULAR  CONTROL  PROBLEMS  WITH  GENERAL 
CONTROL  COSTS  by  J.L.  Menaldi  and  M.  Taksar. 

This  work  deals  with  multidimensional  problems  in  which  a  number 
of  control  functionals  is  different  from  the  dimension  of  the 
underlying  stochastic  process.  Each  coordinate  of  the  stochastic 
process  is  controlled  by  a  linear  combination  of  the  functionals. 

This  is  the  situation  which  arises  in  diffusion  models  of  network  of 
queues . 

2.  SINGULAR  CONTROL  IN  VIDALE-WOLFE  ADVERTISING  MODEL  by  M. 
Taksar  and  S.  Zhu. 

A  singulai.  stochastic  contiol  is  used  to  traet  a  classical 
Vidale-Wolfe  advertising  model  to  find  an  optimal  level  of  production 
and  advertisement  of  a  large  firm  in  a  market  subject  to  saturation. 
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